EnigmA Amiga Run 1998 July/EnigmA AMIGA RUN 29 (1998)(G.R. Edizioni)(IT)[!][issue 1998-07 & 08].iso/recent/poolme.rea

Text File | 1998-06-26 | 23KB | 539 lines

Short:    Memory defragmentizer/AllocP superset
Author:   [e-mail] (Thomas Richter)
Uploader: [e-mail] (Thomas Richter)
Version:  1.40
Type:     util/sys
Requires: OS 3.0

______________________________________________________________________________

        Compatibility notes, other remarks
______________________________________________________________________________

The linux kernal boot program "amiboot" does not work with PoolMem installed.
This program suffers from several bugs. First of all, it examines the
memory list itself, without using the exec.library. (What do you think
AvailMem() is good for, then?) Second, it does not even call Forbid() to
do so. Third, it always expects that the chip memory comes in one continous
block and is too unflexible to handle other cases, as for example the two
ChipMem blocks that will be created when PoolMem is running and active.
If you really MUST use this program, then use the SCRATCHONLY option of
PoolMem to avoid splitting the memory in a large and a small memory pool.
______________________________________________________________________________

The game demo "The Final Odyssey" crashes when PoolMem is active. Tracing
this error down to its roots resulted in an aparent bug in the memory
deallocation function of this game that occurs, too, with MungWall instead
of PoolMem installed. Since EVERY CORRECT PROGRAM is supposed to run with
MungWall, I WON'T TRY to add a workaround for this bug. Just avoid buggy
programs, as usual. Thanks to Dirk Neubauer for the hint.
______________________________________________________________________________

Photogenics: I got reports that this program crashes with PoolMem installed
as soon as it is shut down. A bug in the shutdown code alters memory that
is no longer allocated. THIS IS FATAL, even without PoolMem. It's just a
matter of good luck if that does not crash the system. This hit cannot
be detected with MungWall since pgs called Forbid(). MemSniff will do a
better job in these cases...
______________________________________________________________________________

The mouse driver in the XiPaint program contains a serious bug that can
cause crashes, with or without PoolMem.

The message handling of the mouse driver is broken. After a long debug
session deep in the night, I found that the crashes were created by the
mouse driver XMouse. It replies messages that are no longer valid and have
been FreeMem()'d before. THIS IS DEFINITELY ILLEGAL AND NOT A POOLMEM PROBLEM.

To help you checking other programs as well, I included a tiny patch
"PatchReplyMsg". You shouldn't install it permanently, though, because it
slows down the message system. If, after installation of this patch, a
program crashes with guru 0x01000001f, THEN THIS PROGRAM HAS THE SAME BUG
AS XIPAINT and replies illegal messages. This will crash your system anyways,
with or without PoolMem. Please sent a bug report to the author, or remove
these programs. As I said, THIS IS NO POOLMEM problem.

The guru 0x01000013 is rather typical for these programs - they invalidate
the internal scratch list of PoolMem.
______________________________________________________________________________

To "SaferPatches" users:

To be able to remove PoolMem later on safely, PoolMem must be run AFTER
SaferPatches has been installed. Same goes of course for all other patches
and the SaferPatches program.
______________________________________________________________________________

Developpers should check the "Developper.readme" file in this archive. It
contains information about how to allocate and deallocate memory correctly.
THESE AREN'T POOLMEM SPECIFIC INFORMATIONS! ALL OS FRIENDLY PROGRAMS HAVE
TO OBEY THE RULES FORMULATED IN THIS FILE, they aren't my inventions.
______________________________________________________________________________

It has been reported that PoolMem runs astonishingly slow on PowerUp boards.
I've currently no explanation for this, if you've any idea about it or know
how to fix it - please report!
______________________________________________________________________________

SetPatch, the 68040.library and ppc.boards:

The 68040 library which is loaded by this program seems to allocate the
MMU tables in a strange way incompatible to the scratch list of PoolMem.

Easy solution: run PoolMem after SetPatch. This compatibility problem
*should* have been fixed with the 1.31 version, but hasn't been tested
since I've only a '030 on board. (Sponsors welcome, though... ;-) )
______________________________________________________________________________

MemSniff:    (The THOR MungWall replacement)

Patches deep into the memory allocation routines, too deep for PoolMem.

______________________________________________________________________________

Additional nodes, compatibility hacks invented so far:

o) Programs that rely on a "correctly" set zero condition code of AllocMem()
   ("PixMate")

o) the FFS/OFS ACTION_CLOSE bug that does not set the correct result code.
This is currently done by the PoolMem FreeMem() replacement routine. However,
the author of the FFS ("Heinz") has been informed, so it's likely that this
error will be fixed in the next FFS release, probably for the FFS of the
3.5 workbench.
______________________________________________________________________________

            Version History
______________________________________________________________________________

1.40:

- Integrated partially the PatchRAM AllocVec() patch. Memory requests by the
  RAM disk are now allocated in reverse order. This feature can be turned off
  explicitly by the NORAMREVERSE option, see below.
  Thanks to Raphael Pilarczyk for testing.

______________________________________________________________________________

1.35:

- Added a patch for the AvailMem() function to include the bytes in the
  scratch list.
- Added the ALLOC32 keyword. See below for caveats! You usually DO NOT
  want this!

______________________________________________________________________________

1.34:

- Added the SMALLHEADER keyword. A selected memory block can be set aside
  for the small memory blocks explicitly.
- Fixed a documentation error in the Developpers.readme. Thanks, Dave!

______________________________________________________________________________

1.33:

- Removed a bug in the PoolMem shutdown code. The pre-1.33 releases did not
  free the scratch list completely.
- Added the SCRATCHONLY keyword, it installs only the "scratchlist" of
  PoolMem, not the dynamic pools.
- Added the PUDDLESIZE, PUDDLETHRES and SHRINKTHRES keywords to make the
  parameters of the small memory pool adjustable.
- Rewrote a lot of the kernal routines, mostly for optimizations.
______________________________________________________________________________

1.32:

- Rewrote the startup segment completely. All 1.2 compatible code, argument
  parser etc... removed.
- Added a proper version check. PoolMem requires OS 3.0.
- Added the NOMERGE option to avoid merging of FastMem of different speed.
- As a result of the new argument parser, the CHIPFWD option DOES NO LONGER
  include the "INSTALL" option, both arguments should be given.
- The PoolMem algorithms hasn't been changed, though.
- Included new versions of PatchRAM and ShowMem.
- PoolMem 1.31 and above works fine on PowerUp-Systems. Thanks to Andreas
  Kleinert for testing! However, the current version does not yet adjust
  memory correctly for the PowerUp system, that's still left to the ppc.lib
  and AllocP32. Might change in the future, though.
______________________________________________________________________________

1.31:

PoolMem includes a patch for AllocAbs() to be able to allocate absolute memory
from the scratch list. This *should* help to make PoolMem working with the
68040/68060/ppc library, thus, it should be possible to run PoolMem IN FRONT
of these libraries, i.e. "SetPatch". Would be nice if somebody could sent me
a report whether this works now or not.

Additionally, new versions of the FragMeter and ShowMem program are included.
______________________________________________________________________________

1.30:

This version introduces a major rewrite of all critical routines, especially
the replacement AllocMem() and FreeMem() routine. This might mean that either
old bugs have been fixed or new bugs have been introduced. Anyways, check out
for possible bugs since this is a release with a "0" in the version.
This version might be much faster than the 1.2x version, due to some
optimizations.
WARNING: Since the new 1.30 release of PoolMem replaces the Exec memory
functions completely, YOU MUST RUN POOLMEM IN FRONT OF OTHER MEMORY PATCHES.
ESPECIALLY, RUN POOLMEM FIRST, MUNGWALL AFTERWARS or Mungwall will be non-
functional.

_____________________________________________________________________________

1.22:

PoolMem merges now automatically all adjacent memory lists of the same type.

______________________________________________________________________________

1.21.1:

Included a new version of PatchRAM (check PatchRAM.readme for details),
and a new version of the FragMeter program.
_____________________________________________________________________________

1.21:

PoolMem flushes now the memory as it should if called by AllocMem(-1,..).
Except that, nothing changed.
_____________________________________________________________________________

1.20:

Added the CHIPFWD command. Works like INSTALL but allocates chip mem in
standard (forwards) mode instead of backwards mode. Might solve strange
compatibility problems. Added more sanity checks, included PatchReplyMsg
to create warnings for broken programs.
______________________________________________________________________________

1.19.1:

Changed the name of the DefragMeter to FragMeter because that makes more
sense. Added a chunk count to this program. PoolMem itself unchanged.
______________________________________________________________________________

About PoolMem:

    If you run a lot of programs without resetting the system, you'll
usually find that the main memory of the computer is getting "messed up",
split in lots of tiny memory "snippets" that are more or less useless due to
their tinyness. It may happen that you can't start an application even though
enough memory is available - because this memory is too fragmentated to
be of any use.

That's the point where PoolMem tries to help you: It manages the main memory
in a way such that it can't get fragmentated too easely. It replaces also
the function of AllocP, which is therefore obsolete.
______________________________________________________________________________

Installation and Usage:

    Copy the PoolMem program in this archive to the C: directory of your
system partition. Add the following line to your startup-sequence:

PoolMem INSTALL >NIL:


To remove PoolMem later on, open a shell window and enter

PoolMem remove


The complete command template:

HELP/S,REMOVE/S,INSTALL/S,CHIPFWD/S,NOMERGE/S,
SCRATCHONLY/S,PUDDLESIZE/N,SHRINKTRHES/N,PUDDLETHRES/N
SMALLHEADER/N,ALLOC32/S,NORAMREVERSE/S


HELP:            Prints a small overview of the available options.

REMOVE:            Remove a previously installed version of PoolMem.

INSTALL:        Install the PoolMem patch.

CHIPFWD:        Don't allocate big chunks of chip memory in reverse
            direction.
            NOTE: Starting with 1.32, this is no longer a
            stand-alone option, INSTALL must be given, too.

NOMERGE:        Do not attempt to merge adjacent memory blocks of
            the same type. Even if the memory types are equal,
            the physical characteristics of the memory blocks
            might differ (e.g. 32 bit memory vs. 16 bit memory).
            Future versions of PoolMem might respect this.
            fragmentation.

SCRATCHONLY:        Do to build memory pools for small block allocations,
            but use the scratch list only.

PUDDLESIZE:        Set the "puddle size" for the small block memory pool.
            If an allocation from the small memory pool fails,
            PoolMem attempts to enlarge the pool by this number
            of bytes. Defaults to 8192.

PUDDLETHRES:        Set the "puddle threshold" for the small memory pool.
            Allocations larger than this size are taken from the
            large memory pool instead of the small memory pool.

SHRINKTHRES:        Set the "merge threshold". If more than PUDDLESIZE +
            SHRINKTHRES bytes are unused at the end of the
            small memory pool, the pool shrinks and the free
            bytes enter the large pool. Defaults to 4096.

SMALLHEADER:        If this option is given, set the memory block of this
            index aside and use it as small memory pool. The
            argument is simply the number of the block in the
            memory list, counting from zero, and can be found
            by watching the output of ShowMem. The topmost memory
            block is number 0, the next below 1 and so on.
            This memory block MUST BE fast memory.
            If this option is given, PoolMem does not try to
            split any fast mem header, only the chip mem is
            split.
            The small memory block is sorted in just before the
            chip memory, but is used for the small memory block
            allocations as long as possible.

ALLOC32:        Align all memory allocations to 32 byte boundaries.

NORAMREVERSE:        Do not try to allocate memory for the RAM disk in
            reverse order. Using this option is NOT recommended
            since it may increase the fragmentation.

______________________________________________________________________________

ALLOC32 Caveats:

    This keyword keeps memory allocations aligned to 32 byte boundaries,
    which might be useful if dealing with PPC hardware - by using this
    keyword, the PPC cachelines can't overlap with the MC68K series
    cacheline.

    HOWEVER, using this keyword fragmentizes the memory really badly
    because of the unuseful unaligned boundary snippets that neither
    can be allocated, nor enter the scratch list.

    Even though memory returned from AllocMem() IS aligned to 32 byte
    boundaries, AllocVec'd memory isn't. It is still only long word
    aligned because the first longword of the vector is used to keep
    the size of the allocated memory. This doesn't hurt for the purpose
    this keyword was invented for; it is, even more, REQUIRED to work
    like this because the vector size should be either in the PPC or
    in the MC68K cache, but not in both.

    Due to the heavy fragmentation of memory when using this keyword,
    it is recommended to use it only if absolutely necessary. The
    standard PPC system software deals with the problem of overlapping
    caches correctly so there's by default no need for this keyword.


    This keyword works quite different from AllocP32. It doesn't use
    "magic cookies" in any way and is written in a way compatible to
    the - rather ugly - layers.library allocation methods (check the
    developer.readme file in this archive for details about this). The
    drawback of this compatibility is the rather high fragmentation,
    it's even higher than without PoolMem. You have been warned!
______________________________________________________________________________

            For the advanced user.....
______________________________________________________________________________

Theory of operation:

    PoolMem splits all available memory in two blocks: One block is used
for the small memory snippets (always taken from there), the other block is used
for huge allocations. This partition of the main memory is dynamic, i.e. each
sub-pool can grow and shrink, depending on the memory requirements.

The "public/ANY" memory gets a special treatment. PoolMem manages a "scratch"
list for tiny memory blocks taken from there. Instead of taking these tiny
memory blocks always from the main memory, they are taken from this (global,
though) memory pool and put back into this pool if they get freed. A special
garbage collection task cleans this "scratch list" from time to time, or if
it overruns. The main profit is taken from the layers.library, which uses to
allocate tons of tiny snippets and is therefore the main cause for memory
fragmentation.

The "chip" memory is treated a bit different. It's also split into two
distinct memory pools (small and large), but memory from the large pool
is allocated in reverse direction - unless you run PoolMem with the
ChipFwd keyword, of course; this happens for two reasons: First, it
helps to keep the memory defragmentated, so the big pool can't run that
easely into the small pool. Second, it works around a hardware bug of my
computer (the refresh of the high end chip memory in my computer seems to
be a bit buggy - some bits tend to flip if they aren't frequently accessed,
for example by the DMA processor as display memory.)

For details about the PatchRAM program, check its readme file. As I said, if
you run PoolMem, you're supposed to run PatchRAM as well. It helps PoolMem
a lot in its job! Future versions of PoolMem will include this patch as
well, but the current version is not yet supposed to be "final" in any
way.

_________________________________________________________________________________

Additional programs:

The PoolMem program is still in a somewhat experimental stage, even though
it's running stable for my system for more than two years now - I won't
expect any serious bugs, though.
However, if you like to see how PoolMem works and if it has any effects, I
provided several extra programs:

ShowMem:        Shows the allocated/free memory in a graphical over-
            view. For details, check the ShowMem readme and its
            guide. (Available separately as well)

PatchRAM:        Modifies and improves the RAM disk. Shows the correct
            size of the RAM drive which is then no longer 100%
            full all the time.
            For details, check the PatchRAM.readme.

FragMeter:        Calculates the fragmentation of your memory. The
            output is given separately for each memory type.
            A 100% fragmentation indicates that all the free
            memory is messed up in tiny blocks of eight bytes
            each (maximal fragmentation).
            A Shannon-type approach is used to measure the
            fragmentation (the algorithm calculates the Shannon
            entropy of the memory blocks with the formula
            sum += log(chunk->mc_Bytes/total)
            If you know a better approach for measuring the
            fragmentation, lemme now. This here seems at least
            reasonable for me as a theoretical physicst... ;-)

            This program can be used to test the efficency of
            PoolMem. My measurements indicate that the entropy is
            about halved.

MemoryMess:        A program that tries to fragmentate the main memory
            as bad as possible by allocating and freeing memory
            in random order. Can be used together with the
            FragMeter to measure the efficiency of PoolMem or
            with ShowMem to watch PoolMem at its job. Can be
            canceled safely with ^C (Control-C). Does nothing
            useful except that.

PatchReplyMsg:        Useful to find buggy programs, like XiPaint.
            PatchReplyMsg will create a guru 0x0100001f if a
            program attempts to reply an already de-allocated
            message. Don't install it permanently, but use it
            to find the reason for crashes.


If you've ideas how to improve PoolMem, lemme know....

______________________________________________________________________________

Guru meditations thrown by PoolMem:

0x01000013        Scratch entry illegal.

            Some program invalidated the internal memory scratch
            list of PoolMem - by overwriting memory that has been
            deallocated before. An (in)famous example is XiPaint.
            Run "PatchReplyMsg" and a debugger (e.g. COP) to check
            for details.

0x0100000f        MemHeader not found.

            Some program attempted to free a non-existing block
            of memory. PoolMem wasn't able to locate
            the MemHeader.


0x01000011        MemHeader insane.

            PoolMem found a MemHeader whose number of available
            bytes does not match the size of its pool.


0x01000012        Invalid DeleteHeader (internal).

            Someone tried to deallocate the "large" memory pool.
            This is an internal guru, shouldn't happen.

0x01000014        Memory Pools unordered.

            Someone changed the order of the memory blocks, the
            large memory block is no longer in front of the
            small memory pool.


Additionally, PoolMem throws all standard exec guru's - the >1.3x routines
replace the OS functions completely. As a side effect, PoolMem MUST BE RUN
IN FRONT OF OTHER MEMORY TOOLS, especially RUN POOLMEM FIRST, MUNGWALL LATER.

______________________________________________________________________________

                        The THOR-Software Licence


This License applies to the computer programs known as "PoolMem", "ShowMem",
"FragMeter", "MemoryMess" and "ShowMem".
The "Program", below, refers to such program.


The programs and files in this distribution are freely distributable
under the restrictions stated below, but are also Copyright (c)
Thomas Richter.


Distribution of the Program by a commercial organization without written
permission from the author to any third party is prohibited if any payment
is made in connection with such distribution, whether directly
(as in payment for a copy of the Program) or indirectly (as in payment
for some service related to the Program, or payment for some product
or service that includes a copy of the Program "without charge";
these are only examples, and not an exhaustive enumeration of prohibited
activities). However, the following methods of distribution involving
payment shall not in and of themselves be a violation of this restriction:


(i) Posting the Program on a public access information storage and
retrieval service for which a fee is received for retrieving information
(such as an on-line service), provided that the fee is not
content-dependent (i.e., the fee would be the same for retrieving the same
volume of information consisting of random data).



(ii) Distributing the Program on a CD-ROM, provided that the files
containing the Program are reproduced entirely and verbatim on such
CD-ROM, and provided further that all information on such CD-ROM be
redistributable for non-commercial purposes without charge.



Everything in this distribution must be kept together, in original
and unmodified form.




Limitations.

THE PROGRAM IS PROVIDED TO YOU "AS IS," WITHOUT WARRANTY. THERE IS NO
WARRANTY FOR THE PROGRAM, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT
LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A
PARTICULAR PURPOSE AND NONINFRINGEMENT OF THIRD PARTY RIGHTS. THE ENTIRE
RISK AS TO THE QUALITY AND PERFORMANCE OF THE PROGRAM IS WITH YOU. SHOULD
THE PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF ALL NECESSARY
SERVICING, REPAIR OR CORRECTION.



IF YOU DO NOT ACCEPT THIS LICENCE, YOU MUST DELETE ALL FILES CONTAINED IN
THIS ARCHIVE.

______________________________________________________________________________

Have fun,
    Thomas        June 1998